Use lateral join with subquery on user_statistics table in account_view for network stats#12631
Open
sureshanaparti wants to merge 1 commit intoapache:4.22from
Open
Conversation
…tatistics table in account_view for netstats
Contributor
Author
|
@blueorangutan package |
|
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 4.22 #12631 +/- ##
============================================
- Coverage 17.62% 17.62% -0.01%
+ Complexity 15668 15665 -3
============================================
Files 5917 5917
Lines 531255 531255
Branches 64951 64951
============================================
- Hits 93639 93616 -23
- Misses 427077 427100 +23
Partials 10539 10539
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
Author
|
@DaanHoogland @nvazquez & Others - Should we update MySQL version to 8.0.14 in the compatibility matrix here: https://docs.cloudstack.apache.org/en/4.22.0.0/releasenotes/compat.html ? |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16799 |
Contributor
|
Thanks @sureshanaparti - actually the support for Mysql 8.4 is planned for 4.22.1 (#12417) as 8.0 is reaching EOL in April this year, so I think we can hold this PR until the 8.4 support is verified (and a doc PR won't be needed for 8.0.14 as the compatibility matrix will be updated to 8.4) |
Contributor
in that case we need to update the 4.20 branch as well. |
Contributor
Author
|
@blueorangutan test |
|
@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
14 tasks
|
[SF] Trillian test result (tid-15450)
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes the account_view database view by replacing the separate account_netstats_view with a LATERAL JOIN directly in the main view. The change addresses a performance issue where the MySQL optimizer could not push WHERE predicates through the nested view with GROUP BY, causing full table scans on the user_statistics table. The LATERAL JOIN allows correlated subqueries where predicates can be applied before aggregation, enabling efficient use of the existing UNIQUE KEY on account_id in the user_statistics table.
Changes:
- Replaced
account_netstats_viewreference with LATERAL JOIN subquery inaccount_view - Deleted the
cloud.account_netstats_view.sqlview definition file - Added DROP VIEW statement for
account_netstats_viewin cleanup migration script
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| engine/schema/src/main/resources/META-INF/db/views/cloud.account_view.sql | Replaced left join with account_netstats_view with a LATERAL JOIN containing an inline subquery on user_statistics table |
| engine/schema/src/main/resources/META-INF/db/views/cloud.account_netstats_view.sql | Deleted the view file as it's no longer needed |
| engine/schema/src/main/resources/META-INF/db/schema-42200to42210-cleanup.sql | Added DROP VIEW statement to remove account_netstats_view during upgrade |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+92
to
+100
| left join lateral ( | ||
| select | ||
| coalesce(sum(`user_statistics`.`net_bytes_received` + `user_statistics`.`current_bytes_received`), 0) AS `bytesReceived`, | ||
| coalesce(sum(`user_statistics`.`net_bytes_sent` + `user_statistics`.`current_bytes_sent`), 0) AS `bytesSent` | ||
| from | ||
| `cloud`.`user_statistics` | ||
| where | ||
| `user_statistics`.`account_id` = `account`.`id` | ||
| ) AS `account_netstats` ON TRUE |
There was a problem hiding this comment.
This PR introduces a LATERAL JOIN which requires MySQL 8.0.14 or later. This is a significant change that creates a new minimum MySQL version requirement for CloudStack. This requirement should be documented in release notes, upgrade documentation, and installation documentation to ensure users are aware before upgrading. Consider adding a migration script check or warning if the MySQL version is below 8.0.14.
Suggested change
| left join lateral ( | |
| select | |
| coalesce(sum(`user_statistics`.`net_bytes_received` + `user_statistics`.`current_bytes_received`), 0) AS `bytesReceived`, | |
| coalesce(sum(`user_statistics`.`net_bytes_sent` + `user_statistics`.`current_bytes_sent`), 0) AS `bytesSent` | |
| from | |
| `cloud`.`user_statistics` | |
| where | |
| `user_statistics`.`account_id` = `account`.`id` | |
| ) AS `account_netstats` ON TRUE | |
| left join ( | |
| select | |
| a.id AS `account_id`, | |
| coalesce(sum(`us`.`net_bytes_received` + `us`.`current_bytes_received`), 0) AS `bytesReceived`, | |
| coalesce(sum(`us`.`net_bytes_sent` + `us`.`current_bytes_sent`), 0) AS `bytesSent` | |
| from | |
| `cloud`.`account` a | |
| left join `cloud`.`user_statistics` us ON us.account_id = a.id | |
| group by a.id | |
| ) AS `account_netstats` ON `account_netstats`.`account_id` = `account`.`id` |
kiranchavala
approved these changes
Mar 5, 2026
Member
kiranchavala
left a comment
kiranchavala
left a comment
There was a problem hiding this comment.
LGTM
Before fix
SHOW CREATE VIEW account_view;
| account_view | CREATE ALGORITHM=UNDEFINED DEFINER=`cloud`@`%` SQL SECURITY DEFINER VIEW `account_view` AS select `account`.`id` AS `id`,`account`.`uuid` AS `uuid`,`account`.`account_name` AS `account_name`,`account`.`type` AS `type`,`account`.`role_id` AS `role_id`,`account`.`state` AS `state`,`account`.`created` AS `created`,`account`.`removed` AS `removed`,`account`.`cleanup_needed` AS `cleanup_needed`,`account`.`network_domain` AS `network_domain`,`account`.`default` AS `default`,`account`.`api_key_access` AS `api_key_access`,`domain`.`id` AS `domain_id`,`domain`.`uuid` AS `domain_uuid`,`domain`.`name` AS `domain_name`,`domain`.`path` AS `domain_path`,`data_center`.`id` AS `data_center_id`,`data_center`.`uuid` AS `data_center_uuid`,`data_center`.`name` AS `data_center_name`,`account_netstats_view`.`bytesReceived` AS `bytesReceived`,`account_netstats_view`.`bytesSent` AS `bytesSent`,`vmlimit`.`max` AS `vmLimit`,`vmcount`.`count` AS `vmTotal`,`runningvm`.`vmcount` AS `runningVms`,`stoppedvm`.`vmcount` AS `stoppedVms`,`iplimit`.`max` AS `ipLimit`,`ipcount`.`count` AS `ipTotal`,`free_ip_view`.`free_ip` AS `ipFree`,`volumelimit`.`max` AS `volumeLimit`,`volumecount`.`count` AS `volumeTotal`,`snapshotlimit`.`max` AS `snapshotLimit`,`snapshotcount`.`count` AS `snapshotTotal`,`templatelimit`.`max` AS `templateLimit`,`templatecount`.`count` AS `templateTotal`,`vpclimit`.`max` AS `vpcLimit`,`vpccount`.`count` AS `vpcTotal`,`projectlimit`.`max` AS `projectLimit`,`projectcount`.`count` AS `projectTotal`,`networklimit`.`max` AS `networkLimit`,`networkcount`.`count` AS `networkTotal`,`cpulimit`.`max` AS `cpuLimit`,`cpucount`.`count` AS `cpuTotal`,`memorylimit`.`max` AS `memoryLimit`,`memorycount`.`count` AS `memoryTotal`,`gpulimit`.`max` AS `gpuLimit`,`gpucount`.`count` AS `gpuTotal`,`primary_storage_limit`.`max` AS `primaryStorageLimit`,`primary_storage_count`.`count` AS `primaryStorageTotal`,`secondary_storage_limit`.`max` AS `secondaryStorageLimit`,`secondary_storage_count`.`count` AS `secondaryStorageTotal`,`backup_limit`.`max` AS `backupLimit`,`backup_count`.`count` AS `backupTotal`,`backup_storage_limit`.`max` AS `backupStorageLimit`,`backup_storage_count`.`count` AS `backupStorageTotal`,`bucket_limit`.`max` AS `bucketLimit`,`bucket_count`.`count` AS `bucketTotal`,`object_storage_limit`.`max` AS `objectStorageLimit`,`object_storage_count`.`count` AS `objectStorageTotal`,`async_job`.`id` AS `job_id`,`async_job`.`uuid` AS `job_uuid`,`async_job`.`job_status` AS `job_status`,`async_job`.`account_id` AS `job_account_id` from (`free_ip_view` join ((((((((((((((((((((((((((((((((((((((((`account` join `domain` on((`account`.`domain_id` = `domain`.`id`))) left join `data_center` on((`account`.`default_zone_id` = `data_center`.`id`))) left join `account_netstats_view` on((`account`.`id` = `account_netstats_view`.`account_id`))) left join `resource_limit` `vmlimit` on(((`account`.`id` = `vmlimit`.`account_id`) and (`vmlimit`.`type` = 'user_vm') and (`vmlimit`.`tag` is null)))) left join `resource_count` `vmcount` on(((`account`.`id` = `vmcount`.`account_id`) and (`vmcount`.`type` = 'user_vm') and (`vmcount`.`tag` is null)))) left join `account_vmstats_view` `runningvm` on(((`account`.`id` = `runningvm`.`account_id`) and (`runningvm`.`state` = 'Running')))) left join `account_vmstats_view` `stoppedvm` on(((`account`.`id` = `stoppedvm`.`account_id`) and (`stoppedvm`.`state` = 'Stopped')))) left join `resource_limit` `iplimit` on(((`account`.`id` = `iplimit`.`account_id`) and (`iplimit`.`type` = 'public_ip')))) left join `resource_count` `ipcount` on(((`account`.`id` = `ipcount`.`account_id`) and (`ipcount`.`type` = 'public_ip')))) left join `resource_limit` `volumelimit` on(((`account`.`id` = `volumelimit`.`account_id`) and (`volumelimit`.`type` = 'volume') and (`volumelimit`.`tag` is null)))) left join `resource_count` `volumecount` on(((`account`.`id` = `volumecount`.`account_id`) and (`volumecount`.`type` = 'volume') and (`volumecount`.`tag` is null)))) left join `resource_limit` `snapshotlimit` on(((`account`.`id` = `snapshotlimit`.`account_id`) and (`snapshotlimit`.`type` = 'snapshot')))) left join `resource_count` `snapshotcount` on(((`account`.`id` = `snapshotcount`.`account_id`) and (`snapshotcount`.`type` = 'snapshot')))) left join `resource_limit` `templatelimit` on(((`account`.`id` = `templatelimit`.`account_id`) and (`templatelimit`.`type` = 'template')))) left join `resource_count` `templatecount` on(((`account`.`id` = `templatecount`.`account_id`) and (`templatecount`.`type` = 'template')))) left join `resource_limit` `vpclimit` on(((`account`.`id` = `vpclimit`.`account_id`) and (`vpclimit`.`type` = 'vpc')))) left join `resource_count` `vpccount` on(((`account`.`id` = `vpccount`.`account_id`) and (`vpccount`.`type` = 'vpc')))) left join `resource_limit` `projectlimit` on(((`account`.`id` = `projectlimit`.`account_id`) and (`projectlimit`.`type` = 'project')))) left join `resource_count` `projectcount` on(((`account`.`id` = `projectcount`.`account_id`) and (`projectcount`.`type` = 'project')))) left join `resource_limit` `networklimit` on(((`account`.`id` = `networklimit`.`account_id`) and (`networklimit`.`type` = 'network')))) left join `resource_count` `networkcount` on(((`account`.`id` = `networkcount`.`account_id`) and (`networkcount`.`type` = 'network')))) left join `resource_limit` `cpulimit` on(((`account`.`id` = `cpulimit`.`account_id`) and (`cpulimit`.`type` = 'cpu') and (`cpulimit`.`tag` is null)))) left join `resource_count` `cpucount` on(((`account`.`id` = `cpucount`.`account_id`) and (`cpucount`.`type` = 'cpu') and (`cpucount`.`tag` is null)))) left join `resource_limit` `memorylimit` on(((`account`.`id` = `memorylimit`.`account_id`) and (`memorylimit`.`type` = 'memory') and (`memorylimit`.`tag` is null)))) left join `resource_count` `memorycount` on(((`account`.`id` = `memorycount`.`account_id`) and (`memorycount`.`type` = 'memory') and (`memorycount`.`tag` is null)))) left join `resource_limit` `gpulimit` on(((`account`.`id` = `gpulimit`.`account_id`) and (`gpulimit`.`type` = 'gpu') and (`gpulimit`.`tag` is null)))) left join `resource_count` `gpucount` on(((`account`.`id` = `gpucount`.`account_id`) and (`gpucount`.`type` = 'gpu') and (`gpucount`.`tag` is null)))) left join `resource_limit` `primary_storage_limit` on(((`account`.`id` = `primary_storage_limit`.`account_id`) and (`primary_storage_limit`.`type` = 'primary_storage') and (`primary_storage_limit`.`tag` is null)))) left join `resource_count` `primary_storage_count` on(((`account`.`id` = `primary_storage_count`.`account_id`) and (`primary_storage_count`.`type` = 'primary_storage') and (`primary_storage_count`.`tag` is null)))) left join `resource_limit` `secondary_storage_limit` on(((`account`.`id` = `secondary_storage_limit`.`account_id`) and (`secondary_storage_limit`.`type` = 'secondary_storage')))) left join `resource_count` `secondary_storage_count` on(((`account`.`id` = `secondary_storage_count`.`account_id`) and (`secondary_storage_count`.`type` = 'secondary_storage')))) left join `resource_limit` `backup_limit` on(((`account`.`id` = `backup_limit`.`account_id`) and (`backup_limit`.`type` = 'backup')))) left join `resource_count` `backup_count` on(((`account`.`id` = `backup_count`.`account_id`) and (`backup_count`.`type` = 'backup')))) left join `resource_limit` `backup_storage_limit` on(((`account`.`id` = `backup_storage_limit`.`account_id`) and (`backup_storage_limit`.`type` = 'backup_storage')))) left join `resource_count` `backup_storage_count` on(((`account`.`id` = `backup_storage_count`.`account_id`) and (`backup_storage_count`.`type` = 'backup_storage')))) left join `resource_limit` `bucket_limit` on(((`account`.`id` = `bucket_limit`.`account_id`) and (`bucket_limit`.`type` = 'bucket')))) left join `resource_count` `bucket_count` on(((`account`.`id` = `bucket_count`.`account_id`) and (`bucket_count`.`type` = 'bucket')))) left join `resource_limit` `object_storage_limit` on(((`account`.`id` = `object_storage_limit`.`account_id`) and (`object_storage_limit`.`type` = 'object_storage')))) left join `resource_count` `object_storage_count` on(((`account`.`id` = `object_storage_count`.`account_id`) and (`object_storage_count`.`type` = 'object_storage')))) left join `async_job` on(((`async_job`.`instance_id` = `account`.`id`) and (`async_job`.`instance_type` = 'Account') and (`async_job`.`job_status` = 0))))) | utf8mb4 | utf8mb4_0900_ai_ci |
EXPLAIN ANALYZE SELECT * FROM account_view WHERE id = 4
-> Index lookup on account_netstats_view using <auto_key0> (account_id=4)
-> Materialize (cost=2.35..2.35 rows=3)
-> Group aggregate: sum(user_statistics.current_bytes_sent)...
-> Index scan on user_statistics using i_user_statistics__account_id ← FULL INDEX SCAN
mysql> SHOW CREATE VIEW account_netstats_view;
+-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+
| View | Create View | character_set_client | collation_connection |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+
| account_netstats_view | CREATE ALGORITHM=UNDEFINED DEFINER=`cloud`@`%` SQL SECURITY DEFINER VIEW `account_netstats_view` AS select `user_statistics`.`account_id` AS `account_id`,(sum(`user_statistics`.`net_bytes_received`) + sum(`user_statistics`.`current_bytes_received`)) AS `bytesReceived`,(sum(`user_statistics`.`net_bytes_sent`) + sum(`user_statistics`.`current_bytes_sent`)) AS `bytesSent` from `user_statistics` group by `user_statistics`.`account_id` | utf8mb4 | utf8mb4_0900_ai_ci |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+
1 row in set (0.00 sec)
mysql> SELECT * FROM account_view WHERE id = 4;
+----+--------------------------------------+--------------+------+---------+---------+---------------------+---------+----------------+----------------+---------+----------------+-----------+--------------------------------------+-------------+-------------+----------------+------------------+------------------+---------------+-----------+---------+---------+------------+------------+---------+---------+--------+-------------+-------------+---------------+---------------+---------------+---------------+----------+----------+--------------+--------------+--------------+--------------+----------+----------+-------------+-------------+----------+----------+---------------------+---------------------+-----------------------+-----------------------+-------------+-------------+--------------------+--------------------+-------------+-------------+--------------------+--------------------+--------+----------+------------+----------------+
| id | uuid | account_name | type | role_id | state | created | removed | cleanup_needed | network_domain | default | api_key_access | domain_id | domain_uuid | domain_name | domain_path | data_center_id | data_center_uuid | data_center_name | bytesReceived | bytesSent | vmLimit | vmTotal | runningVms | stoppedVms | ipLimit | ipTotal | ipFree | volumeLimit | volumeTotal | snapshotLimit | snapshotTotal | templateLimit | templateTotal | vpcLimit | vpcTotal | projectLimit | projectTotal | networkLimit | networkTotal | cpuLimit | cpuTotal | memoryLimit | memoryTotal | gpuLimit | gpuTotal | primaryStorageLimit | primaryStorageTotal | secondaryStorageLimit | secondaryStorageTotal | backupLimit | backupTotal | backupStorageLimit | backupStorageTotal | bucketLimit | bucketTotal | objectStorageLimit | objectStorageTotal | job_id | job_uuid | job_status | job_account_id |
+----+--------------------------------------+--------------+------+---------+---------+---------------------+---------+----------------+----------------+---------+----------------+-----------+--------------------------------------+-------------+-------------+----------------+------------------+------------------+---------------+-----------+---------+---------+------------+------------+---------+---------+--------+-------------+-------------+---------------+---------------+---------------+---------------+----------+----------+--------------+--------------+--------------+--------------+----------+----------+-------------+-------------+----------+----------+---------------------+---------------------+-----------------------+-----------------------+-------------+-------------+--------------------+--------------------+-------------+-------------+--------------------+--------------------+--------+----------+------------+----------------+
| 4 | f1450cea-1f65-49ea-a0e7-fdf9c527881f | ACSUser | 0 | 4 | enabled | 2026-02-24 13:06:05 | NULL | 0 | NULL | 0 | NULL | 1 | 9be3d387-117e-11f1-855e-1e00d00002bf | ROOT | / | NULL | NULL | NULL | NULL | NULL | NULL | 0 | NULL | NULL | NULL | 0 | 14 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | NULL | NULL | NULL |
+----+--------------------------------------+--------------+------+---------+---------+---------------------+---------+----------------+----------------+---------+----------------+-----------+--------------------------------------+-------------+-------------+----------------+------------------+------------------+---------------+-----------+---------+---------+------------+------------+---------+---------+--------+-------------+-------------+---------------+---------------+---------------+---------------+----------+----------+--------------+--------------+--------------+--------------+----------+----------+-------------+-------------+----------+----------+---------------------+---------------------+-----------------------+-----------------------+-------------+-------------+--------------------+--------------------+-------------+-------------+--------------------+--------------------+--------+----------+------------+----------------+
1 row in set (0.01 sec)
If user_statistics has 1,000,000 rows
→ MySQL scans ALL 1,000,000 rows, groups them, materializes
→ THEN filters to the 1 account you want
→ O(N) where N = total rows in user_statistics
Index scan (reads entire table/index) → materializes whole account_netstats_view
After fix
SHOW CREATE VIEW account_view;
| account_view | CREATE ALGORITHM=UNDEFINED DEFINER=`cloud`@`%` SQL SECURITY DEFINER VIEW `account_view` AS select `account`.`id` AS `id`,`account`.`uuid` AS `uuid`,`account`.`account_name` AS `account_name`,`account`.`type` AS `type`,`account`.`role_id` AS `role_id`,`account`.`state` AS `state`,`account`.`created` AS `created`,`account`.`removed` AS `removed`,`account`.`cleanup_needed` AS `cleanup_needed`,`account`.`network_domain` AS `network_domain`,`account`.`default` AS `default`,`account`.`api_key_access` AS `api_key_access`,`domain`.`id` AS `domain_id`,`domain`.`uuid` AS `domain_uuid`,`domain`.`name` AS `domain_name`,`domain`.`path` AS `domain_path`,`data_center`.`id` AS `data_center_id`,`data_center`.`uuid` AS `data_center_uuid`,`data_center`.`name` AS `data_center_name`,`account_netstats`.`bytesReceived` AS `bytesReceived`,`account_netstats`.`bytesSent` AS `bytesSent`,`vmlimit`.`max` AS `vmLimit`,`vmcount`.`count` AS `vmTotal`,`runningvm`.`vmcount` AS `runningVms`,`stoppedvm`.`vmcount` AS `stoppedVms`,`iplimit`.`max` AS `ipLimit`,`ipcount`.`count` AS `ipTotal`,`free_ip_view`.`free_ip` AS `ipFree`,`volumelimit`.`max` AS `volumeLimit`,`volumecount`.`count` AS `volumeTotal`,`snapshotlimit`.`max` AS `snapshotLimit`,`snapshotcount`.`count` AS `snapshotTotal`,`templatelimit`.`max` AS `templateLimit`,`templatecount`.`count` AS `templateTotal`,`vpclimit`.`max` AS `vpcLimit`,`vpccount`.`count` AS `vpcTotal`,`projectlimit`.`max` AS `projectLimit`,`projectcount`.`count` AS `projectTotal`,`networklimit`.`max` AS `networkLimit`,`networkcount`.`count` AS `networkTotal`,`cpulimit`.`max` AS `cpuLimit`,`cpucount`.`count` AS `cpuTotal`,`memorylimit`.`max` AS `memoryLimit`,`memorycount`.`count` AS `memoryTotal`,`gpulimit`.`max` AS `gpuLimit`,`gpucount`.`count` AS `gpuTotal`,`primary_storage_limit`.`max` AS `primaryStorageLimit`,`primary_storage_count`.`count` AS `primaryStorageTotal`,`secondary_storage_limit`.`max` AS `secondaryStorageLimit`,`secondary_storage_count`.`count` AS `secondaryStorageTotal`,`backup_limit`.`max` AS `backupLimit`,`backup_count`.`count` AS `backupTotal`,`backup_storage_limit`.`max` AS `backupStorageLimit`,`backup_storage_count`.`count` AS `backupStorageTotal`,`bucket_limit`.`max` AS `bucketLimit`,`bucket_count`.`count` AS `bucketTotal`,`object_storage_limit`.`max` AS `objectStorageLimit`,`object_storage_count`.`count` AS `objectStorageTotal`,`async_job`.`id` AS `job_id`,`async_job`.`uuid` AS `job_uuid`,`async_job`.`job_status` AS `job_status`,`async_job`.`account_id` AS `job_account_id` from (`free_ip_view` join ((((((((((((((((((((((((((((((((((((((((`account` join `domain` on((`account`.`domain_id` = `domain`.`id`))) left join `data_center` on((`account`.`default_zone_id` = `data_center`.`id`))) left join lateral (select coalesce(sum((`user_statistics`.`net_bytes_received` + `user_statistics`.`current_bytes_received`)),0) AS `bytesReceived`,coalesce(sum((`user_statistics`.`net_bytes_sent` + `user_statistics`.`current_bytes_sent`)),0) AS `bytesSent` from `user_statistics` where (`user_statistics`.`account_id` = `account`.`id`)) `account_netstats` on(true)) left join `resource_limit` `vmlimit` on(((`account`.`id` = `vmlimit`.`account_id`) and (`vmlimit`.`type` = 'user_vm') and (`vmlimit`.`tag` is null)))) left join `resource_count` `vmcount` on(((`account`.`id` = `vmcount`.`account_id`) and (`vmcount`.`type` = 'user_vm') and (`vmcount`.`tag` is null)))) left join `account_vmstats_view` `runningvm` on(((`account`.`id` = `runningvm`.`account_id`) and (`runningvm`.`state` = 'Running')))) left join `account_vmstats_view` `stoppedvm` on(((`account`.`id` = `stoppedvm`.`account_id`) and (`stoppedvm`.`state` = 'Stopped')))) left join `resource_limit` `iplimit` on(((`account`.`id` = `iplimit`.`account_id`) and (`iplimit`.`type` = 'public_ip')))) left join `resource_count` `ipcount` on(((`account`.`id` = `ipcount`.`account_id`) and (`ipcount`.`type` = 'public_ip')))) left join `resource_limit` `volumelimit` on(((`account`.`id` = `volumelimit`.`account_id`) and (`volumelimit`.`type` = 'volume') and (`volumelimit`.`tag` is null)))) left join `resource_count` `volumecount` on(((`account`.`id` = `volumecount`.`account_id`) and (`volumecount`.`type` = 'volume') and (`volumecount`.`tag` is null)))) left join `resource_limit` `snapshotlimit` on(((`account`.`id` = `snapshotlimit`.`account_id`) and (`snapshotlimit`.`type` = 'snapshot')))) left join `resource_count` `snapshotcount` on(((`account`.`id` = `snapshotcount`.`account_id`) and (`snapshotcount`.`type` = 'snapshot')))) left join `resource_limit` `templatelimit` on(((`account`.`id` = `templatelimit`.`account_id`) and (`templatelimit`.`type` = 'template')))) left join `resource_count` `templatecount` on(((`account`.`id` = `templatecount`.`account_id`) and (`templatecount`.`type` = 'template')))) left join `resource_limit` `vpclimit` on(((`account`.`id` = `vpclimit`.`account_id`) and (`vpclimit`.`type` = 'vpc')))) left join `resource_count` `vpccount` on(((`account`.`id` = `vpccount`.`account_id`) and (`vpccount`.`type` = 'vpc')))) left join `resource_limit` `projectlimit` on(((`account`.`id` = `projectlimit`.`account_id`) and (`projectlimit`.`type` = 'project')))) left join `resource_count` `projectcount` on(((`account`.`id` = `projectcount`.`account_id`) and (`projectcount`.`type` = 'project')))) left join `resource_limit` `networklimit` on(((`account`.`id` = `networklimit`.`account_id`) and (`networklimit`.`type` = 'network')))) left join `resource_count` `networkcount` on(((`account`.`id` = `networkcount`.`account_id`) and (`networkcount`.`type` = 'network')))) left join `resource_limit` `cpulimit` on(((`account`.`id` = `cpulimit`.`account_id`) and (`cpulimit`.`type` = 'cpu') and (`cpulimit`.`tag` is null)))) left join `resource_count` `cpucount` on(((`account`.`id` = `cpucount`.`account_id`) and (`cpucount`.`type` = 'cpu') and (`cpucount`.`tag` is null)))) left join `resource_limit` `memorylimit` on(((`account`.`id` = `memorylimit`.`account_id`) and (`memorylimit`.`type` = 'memory') and (`memorylimit`.`tag` is null)))) left join `resource_count` `memorycount` on(((`account`.`id` = `memorycount`.`account_id`) and (`memorycount`.`type` = 'memory') and (`memorycount`.`tag` is null)))) left join `resource_limit` `gpulimit` on(((`account`.`id` = `gpulimit`.`account_id`) and (`gpulimit`.`type` = 'gpu') and (`gpulimit`.`tag` is null)))) left join `resource_count` `gpucount` on(((`account`.`id` = `gpucount`.`account_id`) and (`gpucount`.`type` = 'gpu') and (`gpucount`.`tag` is null)))) left join `resource_limit` `primary_storage_limit` on(((`account`.`id` = `primary_storage_limit`.`account_id`) and (`primary_storage_limit`.`type` = 'primary_storage') and (`primary_storage_limit`.`tag` is null)))) left join `resource_count` `primary_storage_count` on(((`account`.`id` = `primary_storage_count`.`account_id`) and (`primary_storage_count`.`type` = 'primary_storage') and (`primary_storage_count`.`tag` is null)))) left join `resource_limit` `secondary_storage_limit` on(((`account`.`id` = `secondary_storage_limit`.`account_id`) and (`secondary_storage_limit`.`type` = 'secondary_storage')))) left join `resource_count` `secondary_storage_count` on(((`account`.`id` = `secondary_storage_count`.`account_id`) and (`secondary_storage_count`.`type` = 'secondary_storage')))) left join `resource_limit` `backup_limit` on(((`account`.`id` = `backup_limit`.`account_id`) and (`backup_limit`.`type` = 'backup')))) left join `resource_count` `backup_count` on(((`account`.`id` = `backup_count`.`account_id`) and (`backup_count`.`type` = 'backup')))) left join `resource_limit` `backup_storage_limit` on(((`account`.`id` = `backup_storage_limit`.`account_id`) and (`backup_storage_limit`.`type` = 'backup_storage')))) left join `resource_count` `backup_storage_count` on(((`account`.`id` = `backup_storage_count`.`account_id`) and (`backup_storage_count`.`type` = 'backup_storage')))) left join `resource_limit` `bucket_limit` on(((`account`.`id` = `bucket_limit`.`account_id`) and (`bucket_limit`.`type` = 'bucket')))) left join `resource_count` `bucket_count` on(((`account`.`id` = `bucket_count`.`account_id`) and (`bucket_count`.`type` = 'bucket')))) left join `resource_limit` `object_storage_limit` on(((`account`.`id` = `object_storage_limit`.`account_id`) and (`object_storage_limit`.`type` = 'object_storage')))) left join `resource_count` `object_storage_count` on(((`account`.`id` = `object_storage_count`.`account_id`) and (`object_storage_count`.`type` = 'object_storage')))) left join `async_job` on(((`async_job`.`instance_id` = `account`.`id`) and (`async_job`.`instance_type` = 'Account') and (`async_job`.`job_status` = 0))))) | utf8mb4 | utf8mb4_0900_ai_ci |
EXPLAIN ANALYZE SELECT * FROM account_view WHERE id = 4
-> Table scan on account_netstats (cost=2.73..2.73 rows=1) (actual time=0.0268..0.0271 rows=1 loops=1)
-> Materialize (cost=0.65..0.65 rows=2) (actual time=0.0261..0.0261 rows=1 loops=1)
-> Aggregate: sum(...) (cost=0.45 rows=1) (actual time=0.0162..0.0163 rows=1 loops=1)
-> Index lookup on user_statistics using account_id (account_id='4')
mysql> SHOW CREATE VIEW account_netstats_view;
ERROR 1146 (42S02): Table 'cloud.account_netstats_view' doesn't exist
mysql> SELECT * FROM account_view WHERE id = 4;
+----+--------------------------------------+--------------+------+---------+---------+---------------------+---------+----------------+----------------+---------+----------------+-----------+--------------------------------------+-------------+-------------+----------------+------------------+------------------+---------------+-----------+---------+---------+------------+------------+---------+---------+--------+-------------+-------------+---------------+---------------+---------------+---------------+----------+----------+--------------+--------------+--------------+--------------+----------+----------+-------------+-------------+----------+----------+---------------------+---------------------+-----------------------+-----------------------+-------------+-------------+--------------------+--------------------+-------------+-------------+--------------------+--------------------+--------+----------+------------+----------------+
| id | uuid | account_name | type | role_id | state | created | removed | cleanup_needed | network_domain | default | api_key_access | domain_id | domain_uuid | domain_name | domain_path | data_center_id | data_center_uuid | data_center_name | bytesReceived | bytesSent | vmLimit | vmTotal | runningVms | stoppedVms | ipLimit | ipTotal | ipFree | volumeLimit | volumeTotal | snapshotLimit | snapshotTotal | templateLimit | templateTotal | vpcLimit | vpcTotal | projectLimit | projectTotal | networkLimit | networkTotal | cpuLimit | cpuTotal | memoryLimit | memoryTotal | gpuLimit | gpuTotal | primaryStorageLimit | primaryStorageTotal | secondaryStorageLimit | secondaryStorageTotal | backupLimit | backupTotal | backupStorageLimit | backupStorageTotal | bucketLimit | bucketTotal | objectStorageLimit | objectStorageTotal | job_id | job_uuid | job_status | job_account_id |
+----+--------------------------------------+--------------+------+---------+---------+---------------------+---------+----------------+----------------+---------+----------------+-----------+--------------------------------------+-------------+-------------+----------------+------------------+------------------+---------------+-----------+---------+---------+------------+------------+---------+---------+--------+-------------+-------------+---------------+---------------+---------------+---------------+----------+----------+--------------+--------------+--------------+--------------+----------+----------+-------------+-------------+----------+----------+---------------------+---------------------+-----------------------+-----------------------+-------------+-------------+--------------------+--------------------+-------------+-------------+--------------------+--------------------+--------+----------+------------+----------------+
| 4 | 30c70a60-6fa1-4553-9ef7-38fcfacae40e | ACSUser | 0 | 4 | enabled | 2026-02-27 05:42:03 | NULL | 0 | NULL | 0 | NULL | 1 | 6b0876f9-139c-11f1-aa13-1e0082000279 | ROOT | / | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | NULL | NULL | 0 | 18 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | 0 | NULL | NULL | NULL | NULL |
+----+--------------------------------------+--------------+------+---------+---------+---------------------+---------+----------------+----------------+---------+----------------+-----------+--------------------------------------+-------------+-------------+----------------+------------------+------------------+---------------+-----------+---------+---------+------------+------------+---------+---------+--------+-------------+-------------+---------------+---------------+---------------+---------------+----------+----------+--------------+--------------+--------------+--------------+----------+----------+-------------+-------------+----------+----------+---------------------+---------------------+-----------------------+-----------------------+-------------+-------------+--------------------+--------------------+-------------+-------------+--------------------+--------------------+--------+----------+------------+----------------+
1 row in set (0.00 sec)
If user_statistics has 1,000,000 rows
→ MySQL directly looks up rows WHERE account_id = 5
→ Aggregates only those rows (e.g., 10 rows)
→ O(k) where k = rows for that specific account
Index lookup with account_id='5' (reads only matching rows) → materializes only for that account
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR uses lateral join (introduced in MySQL 8.0.14) with subquery on user_statistics table in account_view for network stats.
There was a performance issue while querying account_view, which performs full table scans on the user_statistics table, resulting in execution time.
The original account_view used account_netstats_view which aggregates network statistics from the user_statistics table. MySQL's query optimizer could not push WHERE predicates through the nested view with GROUP BY, causing it to materialize the entire derived table before filtering. LATERAL JOIN (supported from MySQL 8.0.14+) allows correlated subqueries where the predicate can be applied before aggregation, avoiding materialization.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
SHOW CREATE VIEW account_view;SHOW CREATE VIEW account_netstats_view;SELECT * FROM account_view WHERE id = 3;EXPLAIN ANALYZE SELECT * FROM account_view WHERE id = 3 \GHow did you try to break this feature and the system with this change?